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I. Introduction 

This is the first report on the evaluation of the Inquiry Based Science and Technology Education 
Program (IN-STEP), an innovative and ambitious science education initiative being undertaken 
by a public -private partnership in Thailand. The IN-STEP project is being sponsored by MSD 
Thailand and supported by funding from its parent company, Merck & Co., Inc., and is being led 
and implemented by a public -private partnership of organizations committed to improving 
science education in Thailand. In response to the tsunami disaster in 2004, Merck and MSD 
Thailand donated US$500,000 to design and implement a science education program in the 
tsunami-affected province of Phang-nga. Managed by the Kenan Institute Asia, IN-STEP draws 
heavily on the experience and resources of the Merck Institute for Science Education (MISE), 
founded by Merck in 1993 to improve student performance in science in the United States. 
MISE’s efforts have shown that the use of well-designed materials and inquiry methods not only 
raises children’s achievement in science, but also their engagement and enthusiasm. 

This report has three aims: first, to share the IN-STEP model with a broad audience of science 
educators and policymakers; second, to provide baseline information on the teachers and schools 
involved in the project; third, to provide the IN-STEP team with useful feedback from the first 
year of implementation. These multiple purposes result in a rather lengthy report. We hope that 
readers will be forgiving and seek out the parts of the report that most interest them. Most of all, 
we hope they will be enticed to follow the progress of IN-STEP over the next few years and 
learn from this effort to adapt strategies used successfully in the United States to improve 
teaching and learning for use in Thailand. 

What Is IN-STEP? 

IN-STEP is intended to improve teaching and learning in science in Thai lower secondary 
schools. The project has six key components: 

• Carefully designed and tested curriculum modules organized around guided 

investigations that illuminate key concepts in the biological, physical, and earth sciences. 
Each module provides 6-8 weeks of instruction. Originally developed and used in the 
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United States, the modules have been translated into Thai and adapted for use in Thailand 
in grades 7, 8, and 9. 

• Intensive professional development experiences for teachers that introduce them to the 
scientific concepts and investigations in each module. Teachers will receive training in 
the use of one new module each year for three successive years. 

• Development of teacher leaders and local capacity to provide professional development 
through the use of accomplished teachers to design and deliver the IN-STEP workshops 
and through the training of accomplished teachers to mentor other teachers. 

• Support for teachers as they implement the modules from mentors and newly developed 
teacher networks intended to foster instructional improvement in science over time. 

• Activities designed to build student and community interest in science such as annual 
science days and science camps that bring scientists and adults using science together 
with students to conduct projects focused on community problems. 

• Careful documentation of the implementation to provide feedback to the IN-STEP team 
members to help them improve the training and implementation as well as an analysis of 
the impact of the project on student performance in science, interest in science, and 
pursuit of science in upper secondary school. 

A central goal of IN-STEP is the introduction of inquiry and associated habits of mind into Thai 
classrooms through the use of the guided investigations in the modules. Inquiry is a fundamental 
aspect of science, and of science education, and it is a central tenet of IN-STEP. Inquiry is a 
systematic intellectual process that great philosophers including Aristotle and Buddha and 
prominent scientists including Thomas Edison and Marie Curie used to gain increased 
understanding of the world around them and develop new knowledge. All too often, however, 
the teaching of science in schools overlooks the inquiry process in order to ensure that students 
learn “science facts” that can be used to score well on examinations, or the inquiry process is 
trivialized into a mechanistic set of procedures — the seven-step scientific method — rather than 
emulating the complex process of investigation, examination of evidence, and reasoning and 
modeling that characterizes the actual work of scientists. The failure to engage students in 
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serious inquiry limits their capacity to understand and “do” science, and ultimately reduces their 
interest in, and access to, careers that depend on scientific knowledge. Lack of experience with 
scientific inquiry and lack of scientific understanding also limit the ability of citizens to apply 
disciplined, rational thinking to their personal lives and to their assessments of their country’s 
science and technology initiatives. Therefore, it is critical for Thailand, and for other nations, to 
improve science teaching by incorporating the inquiry process as a means of helping individual 
students and their communities understand and cope with rapid change and the demands of 
global competition. Data on the performance of Thailand in science relative to other Asian 
nations are presented in Appendix A. 



II. The IN-STEP Theory of Action 

The design of IN-STEP is based on the same theory of action that has guided the work of MISE 
in the United States. This theory involves 12 major elements that are outlined in Eigure 1. The 
first nine elements are already being implemented by IN-STEP and the final three elements will 
be addressed as evidence of the effectiveness of the design is collected over the next two years. 
Each of the 12 elements is discussed briefly below. 

1. Collaborative Design. IN-STEP is a collaboration among numerous agencies including 
MSD-Thailand, MISE, the Kenan Institute Asia (K.I.Asia), the Thai Ministry of 
Education (MOE), the Institute for the Promotion of Teaching Science & Technology 
(IPST), the National Science and Technology Development Agency, the Science Society 
of Thailand, and major universities. Representatives of these organizations have worked 
together to develop a shared vision of good science teaching and have reached agreement 
on the IN-STEP design. By involving key stakeholders in advisory committees both at 
the national and provincial level, significant support has been generated for IN-STEP. At 
the operational level, the IN-STEP team includes professionals drawn from K.I.Asia, 
MISE, MOE, IPST, Teachers College, Columbia, and Thai universities. 
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2. Well-Designed Instructional Materials. These same partners eollaborated on the 
seleetion of earefully designed and tested instruetional materials that emphasize guided 
inquiry and are aligned with Thai national standards. With support from MISE, K.I.Asia 
obtained the rights to use, translate, and adapt seleeted seienee eurrieulum modules 
developed by the National Seienee Resourees Center in the United States. Known as the 
Seienee and Teehnology for Children (STC) materials, each module is organized around 
a major topic in science, such as light, and contains 4-8 weeks of lessons that are 
designed to build student understanding and include investigations that provide 
opportunities for students to apply concepts and do science. The modules contain all of 
the equipment and materials that are needed to teach the lessons and conduct the 
investigations. The translated and adapted versions of the selected modules serve as the 
foundation for IN-STEP. 

Figure 1. The IN-STEP 12-Step Theory of Action 
to Improve Science Teaching and Learning 
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3. Local Engagement. The IN-STEP team selected the province of Phang-nga as the site for 
the pilot program because of the impact of tsunami on the area and the need for 
accelerated economic development. Team members have worked closely with provincial 
leaders to build understanding and support for the program, and the provincial 
educational leaders have played a key role in gaining participation of principals and 
teachers and in setting up the infrastructure to support IN-STEP. 

4. Tapping Expertise in Content and Classroom Practice. Accomplished Thai science 
teachers and content experts from universities and education agencies have been invited 
to serve on the IN-STEP instructional teams charged with designing intensive module- 
based institutes for teachers. Each institute is based on one of the curriculum modules. 
MISE staff have led an annual design retreat to support the work of these instructional 
teams and have led the teams in design and delivery of the initial round of institutes. 

5. Curriculum-Based Professional Development. Beginning in 2007, lower secondary 
science teachers (grades 7, 8, and 9) in Phang-nga were invited to participate in intensive 
professional development for at least three years. Each participating teacher is being 
prepared to teach three modules selected for their grade level. To enable them to make 
the required changes in their teaching practice, they attend an institute focused on one 
new six-week science unit module each year. The instructional teams design and lead 
these institutes at which teachers review and apply the science concepts in the modules, 
participate in the investigations that they will later use with their students, and learn to 
use inquiry-centered instruction. 

6. Implementation Support. IN-STEP has set clear expectations for administrators and 
teachers in participating schools that the new instructional materials and strategies should 
be fully implemented as designed and offers support for implementation from mentors 
and teacher networks. 

7. Developing Science Leaders. Members of the instructional teams and other 
accomplished science teachers have volunteered to serve as mentors to the teachers 
implementing IN-STEP, and these mentors have been provided with training. The 
training and experience provided for instructional team members and mentors are 
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expected to contribute to the development of instructional leaders in science at the 
provincial and school levels who will possess expertise in the curriculum content and the 
capability of providing training and on-site support for other teachers. IN-STEP also is 
encouraging the development of professional learning communities in science by 
networking science teachers, by engaging teachers in common tasks such as assessing 
student work, developing lessons, and mentoring new teachers, and by implementing 
science camps. The IN-STEP mentors are convening teacher meetings in local areas and 
the Educational Service Area Office (ESAO) plans to organize meetings for teachers, that 
it is hoped, will provide the basis for the development of learning communities. These 
activities will build the capacity of the province to sustain the use of the new curricular 
materials and ultimately create capacity to expand IN-STEP to other provinces. 

8. Ensuring Sustained Use of the Materials. The curricular modules contain materials that 
are used up in the course of conducting investigations and, for those modules to be used 
again, these materials must be replenished. To ensure that the modules continue to be 
used and that they are replenished in a cost-effective manner, the provincial education 
office has worked with K.I.Asia to set up a system for collecting, replenishing, and 
distributing the modules. Without this support, modules might be cannibalized for 
materials and eventually cast aside as schools failed to replenish them because of 
inadequate budgets or the difficulty of ordering replacement materials. The provincial 
resource center will help ensure that the investment in the modules continues to pay 
dividends over time. 

9. Formative and Summative Evaluation. An evaluation team is providing regular 
feedback to the developers to improve the implementation and effectiveness of IN-STEP, 
and a rigorous impact study will be conducted to demonstrate its impact on teaching and 
learning. IN-STEP has engaged researchers from CPRE at Teachers College and the 
Institute for the Promotion of Teaching Science and Technology (IPST) to work together 
with K.I.Asia staff to conduct a timely and rigorous evaluation. In addition to 
documenting the use of the IN-STEP materials and their effects on student learning, IN- 
STEP will track the students’ choices of subjects in high school and their participation in, 
and success at, carrying out projects that contribute to the improvement of their 
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communities’ economic development. The involvement of Thai researchers in the 
evaluation also will contribute to their capacity to conduct future implementation studies. 

10. Clear Indicators of Success. The activities and resources described above are part of a 
coordinated strategy to improve student learning in science and create more positive 
attitudes among students about science, science learning, and science careers. The key 
indicators of success for IN-STEP will be increases in the number of students pursuing 
science studies in upper secondary schools, improved performance on national and 
international assessments in science, increased participation in science competitions, and 
more positive attitudes toward science among students and community leaders. 

11. Community Engagement and Support. IN-STEP seeks to increase community support 
for science education and to enhance student motivation to study science by providing 
activities that bring educators, students, and community members together to do science. 
Community members, teachers, and students have enjoyed the province-wide science day 
events led by MSD volunteers. Students have been invited to participate in science camps 
where they learn how to apply their science knowledge to projects that are relevant to 
their local contexts. These will be annual events in the province. The IN-STEP team will 
document and share student projects aimed at contributing to the improvement of the 
quality of life in their communities. Community members with special expertise are 
being recruited to share their knowledge and provide continued support for students. By 
engaging staff from national parks, local universities, and private organizations in 
designing and implementing the science camps, IN-STEP will leverage the expertise and 
resources in the local area to help build a true learning community in science. 

12. Developing Capacity to Sustain and Scale the Reforms. A major goal is to develop 
capacity to expand the program to other schools in other provinces and to other grade 
levels and to share the strategies, and evidence of their effectiveness with leaders at all 
levels — school, provincial, and national — in order to create support for expansion of the 
program if warranted by the evidence. Over three years, IN-STEP will develop a cadre of 
experienced trainers and mentors who will be capable of extending the program to other 
provinces without external assistance. IN-STEP also has provided opportunities for staff 
from MOE and IPST to participate in the training. 
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This is the IN-STEP plan. It is a plan based on research and experience in the U.S context, but it 
is being implemented in a different setting, a setting in which the resources, practices, beliefs, 
and values differ. And while the model has been adapted to fit Thailand, experience may reveal 
that further adjustments are needed to produce the results that IN-STEP’ s designers aspire to 
achieve. This is why the model is being piloted in one province, and why it is being carefully 
evaluated. 

Year One (2007) in Brief: Building the IN-STEP Partnership 

While the design work began in late 2006, the first full year of the project was 2007. Once MSD 
Thailand, MISE, and the Kenan Institute Asia had agreed on the broad parameters of IN-STEP, 
the next step was to develop a broader partnership. Obtaining commitment and engagement from 
provincial and national government leaders and leading science educators were the first critical 
steps. After conducting a series of face-to-face meetings with leaders in science education and 
establishing an advisory board and technical working groups to provide mechanisms for partners 
to participate, IN-STEP gained strong support from the Ministry of Education, the Institute for 
the Promotion of Teaching Science and Technology (IPST), the governor of Phang-nga, directors 
and educational supervisors from the provincial education office in Phang-nga, and principals, 
master teachers, and teachers from Phang-nga schools. 

The next step was to secure inquiry-based curriculum materials. MISE found a willing partner in 
the National Science Resources Center (NSRC), a joint venture of the Smithsonian Institution 
and the National Academy of Sciences in the United States. NSRC had developed a set of 
curriculum modules known as Science and Technology for Children (STC) that were widely 
used in the United States and also in other countries. NSRC granted K.I.Asia the right to 
translate, adapt, and use these materials in Thailand. A group of Thai science educators including 
senior representatives from MOE and IPST attended an NSRC strategic planning workshop and 
reviewed the materials, selecting nine modules that matched the Thai standards for the lower 
secondary grades. Their recommendations were reviewed and approved by MOE, IPST, and 
MISE. K.I.Asia arranged for the translation and adaptation of the teacher and student manuals 
and other materials in the modules. The initial modules being included in IN-STEP are listed in 
Appendix B. 
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To engage local stakeholders and students, IN-STEP organized a Fun Science Day in partnership 
with the Educational Service Area Office of Phang-nga in September, 2006, to introduce inquiry- 
based learning using fun science games, a mobile planetarium, and science experiments. A total 
of 360 students from 48 schools in Phang-nga attended the event. 

Following the MISE model, the IN-STEP team recruited accomplished science teachers and 
science educators to serve as members of the instructional teams that would develop and deliver 
the workshops for teachers. These teachers taught the modules in their classrooms to gain first- 
hand experience with the materials. In October, 2006, these teachers and their principals attended 
a workshop on inquiry, followed by a design retreat led by MISE staff that allowed the 
instructional team members to gain more experience with the modules and investigations and 
then to plan workshops for the teachers. 

After nine months of preparation, IN-STEP was officially launched on March 26, 2007, with the 
signing of a Memorandum of Understanding by the representatives of the Ministry of 
Education’s Office of the Basic Education Commission (OBEC), The Institute for the Promotion 
of Teaching Science and Technology, MSD (Thailand) Etd., and the Kenan Institute Asia. 

Immediately following the launch ceremony, the first IN-STEP professional development 
workshop was conducted for 43 participating science teachers from 22 schools in Phang-nga. 

The five-day workshop was conducted by 18 accomplished teachers who worked with six 
instructional team leaders from MISE. 

The MISE staff worked closely with their Thai counterparts to provide effective learning 
experiences for the 43 participating teachers, allowing them first to experience the inquiry 
process themselves by conducting the investigations in a module and then to practice how to 
engage their students in these same investigations. After the completion of the training, some 
members of the instructional teams provided continuing assistance to their fellow teachers by 
serving as mentors, helping them with the use of modules and with the improvement of their 
science teaching. 

“This workshop is very different from other workshops I have attended throughout my whole 
life,” said Ajam Jedsada Sriwiset, science teacher and academic head of Kho Yao Wittaya 
School. “I have learned many techniques which are impressive and very valuable.” After 20 
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years of teaching and participation in many workshops, Ajarn Jedsada said that he had “learned a 
lot in different way,” noting, “I did not quite understand why I did not get the answer right away 
until I realized that it is not the answer that is important, but the process of finding out the 
answer.” He added that he would apply what he learned to his students. “It’s very important to 
ask the right question at the right time and let students to think about it rather than tell them the 
answer right away.” 

Year Two (2008) in Brief: Initial Implementation 

In 2008 IN-STEP will be expanded, and another cadre of 45 teachers will be prepared to teach 
one of the original three curriculum units — one each for grades 7, 8, and 9 — and the original 43 
teachers will attend workshops preparing them to teach a new module for their grade level. 
Instructional teams have met to revise and deliver workshops for the original three units, as well 
as to design workshops for the three new units. These six workshops will be offered to teachers 
in April, 2008. Implementation of the new curriculum units will again be supported by volunteer 
mentors, electronic communications and tools, and provincial teacher meetings. 

It is hoped that IN-STEP will become a model for Thailand for the improvement of Thai 
students’ performance in science. In the long term, IN-STEP’ s goal is to contribute to Thailand’s 
international competitiveness in science and technology. Ultimately, with this higher capability, 
the Thai people will be able to apply their knowledge of science and technology to preserve the 
country’s natural resources and to live in harmony with nature. 



III. The Evaluation of IN-STEP 

The Consortium for Policy Research in Education (CPRE) at Teachers College, Columbia 
University, has been asked to lead the design and conduct of an evaluation of IN-STEP. CPRE 
includes researchers from seven universities in the United States, and the IN-STEP evaluation 
team includes researchers from two of these institutions. Teachers College, Columbia University, 
and the University of Pennsylvania. The CPRE researchers are collaborating with research staff 
from K.I.Asia and IPST to conduct the evaluation. 
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The central questions to be answered by the evaluation are these: 

• What is the impact of the IN- STEP curriculum materials and training on teachers’ 
classroom practice? 

• What is the impact of the changes in practice on student learning in science? 

• What factors affect the implementation, sustainability, and spread of the IN-STEP 
concept of good science teaching? 

There are three major tasks facing the evaluation team: (1) documenting the instructional team 
retreats and the teacher workshops to give timely feedback to MISE and K.I.Asia staff and the 
IN-STEP instructional teams; (2) documenting the use of the modules by the teachers 
participating in IN-STEP and obtaining feedback from these teachers to help K.I.Asia and MISE 
strengthen the program and ensure full implementation; and (3) assessing the impact of IN-STEP 
on teachers’ classroom practice and on student learning in science. These three tasks are 
discussed here in reverse order. 

The Impact Evaluation. Measuring the impact of IN-STEP requires overcoming four serious 
challenges. Eirst, the evaluation team must identify a reliable measure of science learning that is 
sensitive to the content of the IN-STEP modules. Second, this test must be administered by all of 
the participating schools in a consistent manner. Third, the team must estimate the effects on 
student learning without using a comparison group because it appears to be difficult (but not 
impossible) to construct a credible matched comparison group of classrooms or schools given the 
limited data available nationally on students in Thailand. Einally, the evaluation team must 
persuade stakeholders to be patient as the modules must be fully implemented before a good 
measure of the effects of IN-STEP can be obtained. Since teachers are only implementing one 
six- week module per year, the effects of IN-STEP are likely to be modest in the first and second 
years, even if everything has gone well. By the third year, some teachers will be using three 
modules, which will constitute about 20 weeks of instruction or about half of the school year, 
and it would be reasonable to expect observable effects on learning. 

The best answer to the first two questions would be to use the 9* grade national science test, but 
there is no reason to believe that this test will contain sufficient items on the content covered by 



11 




IN-STEP: Evaluation of the Eirst Year 



the IN-STEP modules to be sensitive to the effective use of the modules. As an alternative, IPST 
has offered to develop a science assessment for use in Phang-nga that will be sensitive to this 
content. This test will provide a good measure of the impact of the program on student learning. 
This will solve the first two problems described above. The third problem will be addressed by 
examining differences across students in the province who have been exposed to one, two, or 
three modules or none of the modules, which will allow the CPRE researchers to develop 
relatively rigorous estimates of the effects of the program. The fourth problem will be addressed 
by providing stakeholders with good data on implementation in 2007, 2008, and 2009 while 
reminding them that the impact evaluation will be forthcoming. 

Feedback on Implementation. Tracking implementation is normally labor-intensive and 
therefore relatively costly. Inexpensive approaches such as teacher surveys that rely wholly on 
self-reporting are notoriously biased as the respondents often know what answers are desired and 
provide the socially acceptable response. But measuring the degree of implementation is 
important both for the impact analysis and for providing feedback to the training teams to 
improve the program. The evaluation team has proposed three ways of doing this that are 
relatively inexpensive. Eirst, the follow-up mentors have been asked to file site-visit reports for 
each school, which will provide answers to some basic implementation questions. Since they are 
already visiting the schools, this will not be an added burden. Concerns about error introduced by 
favoritism will be addressed by comparing these reports to the results from telephone interviews 
with a sample of teachers. Concerns about workload for the mentors are being addressed by 
keeping the forms simple. 

A second approach to measuring classroom implementation is to conduct telephone interviews 
with teachers during the period in which they are using the modules. The IPST volunteers and 
K.I.Asia staff conducted these interviews this year. CPRE staff provided training and assisted 
with the development of protocols and coding of the data. With additional assistance from 
CPRE, K.I.Asia staff have entered these data into Atlas Ti and analyzed the data for this report. 
Similar interviews will be conducted in the Summer and October, 2008, but after the number of 
teachers increases in 2009, only samples will be interviewed. Similar interviews were conducted 
with the IN-STEP mentors, and these data also were analyzed for this report. These interviews 
will be repeated in 2008. 
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A third option would be to conduct a survey of the teachers, and if a survey is used in the future, 
it will be used in conjunction with site visit reports and interviews in order to make adjustments 
resulting from teachers’ tendency to overestimate their use of reform practices. 

Documenting the Training. This has been the most straightforward part of the evaluation. The 
interviewers from K.I.Asia and IPST who were trained by CPRE staff to conduct teacher 
interviews also asked questions about the workshops. Again, CPRE staff helped with the 
protocols and with the coding of the data. In addition to this, participants in both the instructional 
team retreats and IN-STEP workshops were asked to fill out simple surveys at the end of these 
experiences. The data collection efforts and the response rates are displayed in Table 1. Note that 
with the exception of the reports from the mentors, the response rates are uniformly high. 



Table 1. Data Sources for the IN- STEP Evaluation 



Source 


Total Participants 


Response Rate 


Survey at Instructional Team Retreat, October 2006 


28 


93% 


Interviews With Teachers at IN-STEP Workshop, March 2007 


43 


100% 


Survey of teachers at IN-STEP Workshop, March 2007 


43 


100% 


Survey of Mentors, Mentor Training, June 2007 


18 


100% 


Survey of Mentors, Mentor Training, October 2007 


17 


100% 


Telephone Interviews With Teachers, October 2007 


43 


100% 


Reports Prom Mentors, October 2007 


17 


53% 



Organization of the Report 

This is the first report produced by the evaluation team, and presents baseline data collected in 
the first year plus some initial analyzes that might provide insight into how to strengthen the 
program, especially the supports for implementation, and what the challenges are likely to be in 
the coming two years. The data sections are organized in five parts. Section IV focuses on the 
characteristics of the schools and teachers participating in IN-STEP so far. Section V examines 
the composition and experience of the instructional teams who serve as the designers and 
instructors for the teacher professional development. Section VI examines how various 
participants — teachers, instructional team members, and mentors — perceive and assess the 
preparation they have received from IN-STEP. Section VII presents data on the use of the 
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curriculum materials by the first cadre of teachers. Section VIII examines the experience of the 
mentors in their initial efforts to support teachers. And Sections IX and X summarize the impact 
of IN-STEP to date as well as the findings from this first round of data collection, discuss the 
challenges that lay ahead, and offer some recommendations to the IN-STEP partners. 

IV. The IN-STEP Schools and Teachers 

In this section we take a look at the schools and teachers who are participating in IN-STEP and 
report some baseline information that might help us account for the observed patterns of 
participation and implementation reported on in Sections VII. What are the primary 
characteristics of the schools and teachers who have volunteered to participate in IN-STEP, and 
how do they vary? With respect to the schools, we look at their location, their size, the nature of 
their professional communities, and the past performance of their students in science. With 
respect to the teachers, we examine their age, preparation in science, recent professional 
development, why they are participating and their expectations, and who they go to now for 
support in science. 

The Schools. Eifty-three schools serve lower secondary students in Phang-nga. The map in 
Appendix C shows where the schools are located in the province. While few of the schools can 
be described as isolated, they are scattered across a large area. Given the shape of the province, 
traveling from the north to the south of the province takes about D-/i hours. This makes it difficult 
to call a meeting in a central location and expect all of the teachers or principals to show up. 
When teacher meetings are held, they must be organized regionally to limit the amount of travel 
for participants. 
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Table 2. Enrollment in Lower Secondary Schools: First-Year Participants 

and Schools Province-Wide 



Enrollment in Grades 7-9 
(Number of students) 


Schools Sending 
Teachers in Year 1 


All Schools 
in the Province 


Small School 


<120 students 


3 


27 


Medium School 


121 - 300 students 


10 


17 


Large School 


301 - 600 students 


3 


3 


Very Large 


>601 students 


6 


6 




Total 


22 


53 



Source: Phang-nga Educational Service Area Office. 



Regarding school size, Table 2 shows that during the first year of implementation, all nine of the 
large or very large schools in the province were participating in IN-STEP. This means that in the 
second year and third years of the pilot, more teachers from medium and small schools will join 
the program. Because these teachers have fewer colleagues in science — in some cases, there is 
only one science teacher in a school — and because they might be assumed to be more reluctant 
participants in that they did not volunteer in the first phase, they likely will require higher levels 
of support to implement the new curriculum units. 

Moreover, while school size is inversely related to school performance in the United States — a 
finding that has led to the contemporary small-schools movement — it is the opposite case in 
Thailand. Small schools are usually rural schools that are under-resourced and attract poorer 
students, and their results on examinations reflect these conditions (Bank of Thailand, 2006). 
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Table 3. The Average General Achievement Test (GAT) Scores of 9th Grade Students 
in Phang-nga: Compared With National Scores, 2003-2006 



Subject 


2003 


2004 


2005 


2006 


Phang- 

nga 


Nation 


Phang- 

nga 


Nation 


Phang- Nation 
nga 


Phang- 

nga 


Nation 


Thai 


54.07 


53.98 


54.07 


38.29 


42.96 


43.24 


43.94 


Math 


33.66 


34.99 


34.93 


34.88 


31.25 


29.31 


31.15 


Social 


50.12 


49.33 


43.91 


42.44 


38.44 


42.09 


41.69 


Studies 
















Science 


37.16 


38.07 


37.93 


37.22 


39.77 


39.74 


39.34 


English 


36.32 


37.92 


31.34 


32.27 


28.51 


30.21 


30.85 


Average 


- 


- 


37.28 


37.02 


36.19 


36.92 


37.39 



Source: Phang-nga Educational Service Area Office. 

The GAT scores of Phang-nga students are comparable to the national averages in all subjects. 
This means that the schools in Phang-nga are rather representative of schools in Thailand as a 
whole, and this makes the province a good place to pilot new curricula. This is especially true in 
terms of student performance in science. From Tables 3 and 4, it is clear that performance trends 
in mathematics and science in the province have not shown any improvement from 2004 to 2006. 
In fact, mathematics performance has declined relative to national performance. 
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Table 4. Comparison of GAT Scores 2004-2006 of 9th Grade Students in Phang-nga 




Thai Math Social Study Science English 

Source: Phang-nga Educational Service Area Office. 



□ 2004 

□ 2005 
■ 2006 



The data in Table 5 show that the schools currently participating in IN-STEP represent a rather 
broad distribution of achievement. The national average score on the GAT 9* grade science test 
in 2006 was 39.34, so students in the high and medium-high performing schools are scoring on 
average above the national average on the test, and those in the other two categories are scoring 
below the national average. 



Table 5. School-Level Performance on the 9th Grade GAT Science Assessment 



Performance Categories 


Schools Sending Teachers 
in Year 1 


All Schools 
in the Province 


High (GAT score >45) 


4 


9 


Medium-High (40-44.99) 


7 


10 


Medium-Low (35-39.99) 


7 


22 


Low (<35) 


4 


10 


Totals 


22 


51* 



*Note: Missing data for two schools. 
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Table 6. Distribution of IN-STEP Teachers 
by Level of Faculty Participation (Penetration) and School Size 







Size of the Schools 




Participation 
in IN-STEP 


Small 


Medium 


Large 


Very 

Large 


Total 


All science 
teachers in the 
school 


1 


3 


0 


3 


7 


50% or more 


3 


10 


5 


3 


21 


Less than 50% 


0 


3 


3 


9 


15 


Total 


4 


16 


8 


15 


43 



Source: March, 2007, teacher interviews. 



From the data presented in Table 6, we see most of the participating teachers are from school 
settings where other teachers are participating in IN-STEP. As more teachers enroll from small 
schools, this will be less likely to be the case. This suggests that the IN-STEP team should 
consider the merits of alternative rollout strategies. We discuss these in Section X. 

Table 7. Teachers' Perceptions of Principal Support 
for Teaching and Learning in Science 



Principals' Support 



Module 



Participation 


Earth in 
Space 
(EIS) 


Energy, 

Machines, 

Motion 

(EMM) 


Light 

(L) 


Total 


Highly Supported 


5 


8 


5 


18 


Supported 


4 


2 


6 


12 


Neutral 


2 


3 


2 


7 


Unsupported 


1 


0 


2 


3 


Highly Unsupported 


0 


2 


0 


2 


No Response 


1 


0 


0 


1 


Total 


13 


15 


15 


43 


Source: March, 2007, teacher interviews. 
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The data on teachers' perceptions of principals' support in Table 7 are highly positive. About 
70% of the participating teachers think that their principals will support them in implementing 
IN-STEP. For example, one respondent said, “The principal provides good support always since 
he has a science background. Science has a higher budget than other subjects in my school” (IN- 
STEP field notes, October, 2007). Another said: “Sometimes, the principal comes to observe my 
class. Mostly, if I request something important for teaching, I will get support” (IN-STEP field 
notes, October, 2007). These comments were typical, but 28% of teachers expressed concerns 
about how much support they were receiving or were likely to receive from their principals. 
They gave responses such as this: “[My] principal does not give priority on science. Mostly, he 
supports activities and music since it can promote school to public.” Or: “There [is] not enough 
equipment in science . . . .The principal rarely visits my class . . . .” (IN-STEP field notes, October, 
2007). An exploration of the sources of these concerns is required in order to find effective ways 
to enhance principal support for IN-STEP. The relationship between the level of a principal’s 
support and the level of a teacher's implementation will be tracked over time, because it is 
anticipated that this will affect the level of initial implementation and sustainability over time. 



Table 8. Teachers' Perceptions of Professional Community 
in Their Schools by Training Group 



Level of Teacher 
Collaboration 




Module 




Total 


EIS 


EMM 


L 


High Collaboration 


3 


1 


1 


5 


Some Collaboration 


5 


3 


2 


10 


Weak Collaboration 


3 


9 


5 


17 


No Collaboration 


2 


2 


6 


10 


N/A 


0 


0 


1 


1 


Total 


13 


15 


15 


43 



Source: March, 2007, teacher interviews. 

Sixty-four percent of the teachers reported that collaboration among teachers in their schools was 
weak. (By “weak,” we mean that teachers rarely meet formally and almost never work together 
on any aspect of instruction.) In fact, almost a quarter of the teachers reported that there was no 
communication among science teachers and some said they liked it that way. By contrast, we 
categorized responses indicating regular meetings and sharing of equipment and materials, but 
no collaboration on instruction tasks, as "some collaboration," and cases in which teachers 
reported frequent cooperation on lesson or unit planning, joint design of assessments and scoring 
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of student work, and observing each other teaching as "high collaboration." These data suggest 
that it will be difficult to create cultures of collaboration in the schools, especially given the 
limited resources available for on-site support. 

In our judgment, it might be wiser for the IN-STEP team to focus on developing teacher 
networks in several regions of the province. While it will be challenging to develop an effective 
mechanism for supporting sustainable networks that fit with local schools’ cultures, it could be 
critical for sustaining the use of the modules and inquiry methods. In many schools, there is only 
one science teacher teaching at each grade level, so the teacher has no colleagues who are using 
the same modules and therefore no one to talk to about engaging students, student 
misconceptions, or other problems he or she might encounter in the classroom. A network can 
provide this kind of support and also can serve as a means of improving practice over time as 
teachers share strategies that appear to be successful. The networks also will expose teachers to 
the benefits of collegial interaction and might contribute to changes in the school cultures 
themselves. Assigning accomplished teachers or mentors as lead teachers in a region, providing 
them with small stipends and support from principals and ESAO staff, and asking them to plan 
and lead teacher meetings might result in sustainable teacher networks. The main point here is 
that IN-STEP is not positioned to overcome the obstacles to teacher collaboration that exist in 
many of the schools and needs to develop an alternative strategy to support instructional change. 



Table 9. Colleagues' Attitudes Toward IN-STEP 



Feelings About IN-STEP Participation 


EIS 


Module 

EMM 


L 


Total 


Positive 


9 


9 


12 


30 


Neutral 


2 


3 


1 


6 


Negative 


0 


0 


0 


0 


Mixed 


2 


2 


0 


4 


N/A 


0 


1 


2 


3 


Total 


13 


15 


15 


43 



Source: March, 2007, teacher interviews. 

The data in Table 9 show that 10 out of 43 or 23% of the teachers mentioned that their 
colleagues have neutral or mixed attitudes towards IN-STEP, suggesting that there is still some 
work to be done to make teachers aware of the benefits of IN-STEP. 
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The Teachers. The IN-STEP teachers are experienced. Table 10 shows that 86% of them have 
more than three years of teaching experience and half of them have over 10 years of teaching 
experience. Their age distribution is quite dispersed, covering from under 30 to over 50. It might 
be interesting to explore the relationship of teachers’ age and teaching experience, their openness 
to new teaching practices, and the levels of implementation observed in their classrooms. 



Table 10. Age and Teaching Experience of IN-STEP Participants 









Age 






Under 30 


31-40 


41-50 


Over 50 


Total 


Under 3 years 


3 


2 


NA 


1 


6 


3-10 years 


5 


7 


1 


NA 


13 


Over 10 years 


NA 


3 


10 


11 


24 


Totals 


8 


12 


11 


12 


43 



Source: March, 2007, teacher interviews. 

All of the IN-STEP teachers majored in science at the university or teacher training college. This 
implies that most of them should have relatively strong content knowledge. However, as Table 
1 1 shows, almost half of the participants majored in general science and some might be teaching 
out of field, for example, teaching a course in physics when their degree is in biology. So in 
future analyses we should examine the relationship between teachers’ science background and 
their implementation of modules in specific domains of science. 
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Table 11. Teachers’ Formal Science Background Content and Credentials 



Field 


Undergraduate 

Major 


Graduate Degrees 


Biology 


6 


- 


Chemistry 


7 


- 


Physics 


6 


- 


General Science 


21 


- 


Other 


2 


1 


Totals 


42 


1 



Source: March, 2007, teacher interviews. 



Table 12 shows that teachers have had rather diversified professional development experiences 
in both science-related and nonscience subjects. While our data do not tell us the precise content 
of these experiences, the topics suggest a stronger emphasis on methods and technology than on 
science content or how students learn science. It might be interesting to explore the teachers’ 
perceptions of the differences between their past training and IN-STEP. This will contribute to 
increased understanding of the methodology IN-STEP uses to develop high-quality professional 
development. 
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Table 12. Previous Professional Development of IN-STEP Teachers 



Field Number 

Instructional Design (Lesson Planning) 16 

Inquiry-Based Learning 6 

Computer Literacy (ICT) 3 

Chemistry 2 

Child-Centered Teaching 2 

Environmental Science 2 

Instructional Media 2 

None (IN-STEP is the first) 2 

Other (Assessment, Biology, Curriculum and Assessment, Electronics, Geography, 8 

Management, Tsunami, and Video Production) 



Source: March, 2007, teacher interviews. 

Table 13. Nature of the Teachers' Decisions to Participate 



Nature of Decision 


EIS 


Module 

EMM 


L 


Total 


Volunteered 


5 


8 


6 


19 


Assigned but Willing to Attend 


8 


5 


7 


20 


Assigned but Reluctant to Attend 


0 


2 


2 


4 


Total 


13 


15 


15 


43 



Source: March, 2007, teacher interviews. 



Ninety percent of the teachers indicated that they were willing to attend the workshop (see Table 
13), and over 88% of teachers (38 of 43) mentioned that they expected to acquire new teaching 
techniques or improve their techniques by participating in IN-STEP (see Table 14). Other 
frequently mentioned expected benefits included acquiring new materials and increasing science 
knowledge. Next year we will revisit this information to see if teachers feel they were able to 
meet their expectations over the two years of implementation and to probe more deeply into what 
really motivates teachers and works for them. We also will probe into why some teachers were 
reluctant to attend, and what we learn might lead to better ways to communicate about IN-STEP. 



23 







IN-STEP: Evaluation of the Eirst Year 



Table 14. What Teachers Expect to Gain 



Expected Benefits 




Module 






(Number of Mentions) 


EIS 


EMM 


L 


Total 


New instructional materials 


3 


5 


9 


17 


Increased personal knowledge of science 


4 


9 


2 


15 


New teaching techniques 


12 


12 


14 


38 


More active students 


7 


4 


5 


16 


Improved student learning of science 


2 


4 


3 


9 


Use of IN-STEP materials in other grades 


2 


2 


2 


6 


Networking/new friends 


0 


1 


1 


1 


Improved assessment strategies 


0 


1 


0 


1 


Total 


30 


38 


36 


104 


Source: March, 2007, teacher interviews. 










From Table 15, it is clear that many teachers receive little instructional support and rely mostly 
on themselves. Others seek support primarily from other teachers within their school or in other 


schools. These data are in agreement with those from the previous table showing that support 
within the professional community is very low. It is noteworthy that teachers do not report 




principals or external experts such as university faculty 


as sources of support. This may be a 




function of how we asked the questions, and in future interviews we will explore what kind of 


support teachers feel they need, and what they seek from other teachers in order to properly 




design an effective support system. These data show that there is 


an opportunity for IN-STEP to 


provide training or a mechanism to fill an important gap. 








Table 15. Sources of Instructional Support 










Module 






Participation 


EIS 


EMM 


L Total 


Rely on myself 


7 


13 


12 


32 


Lead teacher 


3 


4 


3 


10 


Teachers in my school 


10 


9 


9 


28 


Teachers in other schools 


3 


4 


5 


12 


Other experts (university professors, etc.) 


2 


2 


5 


9 


Principal 


0 


0 


0 


0 


ESAO supervisors 


0 


0 


0 


0 


Total 


25 


32 


34 


91 



Source: March, 2007, teacher interviews. 
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Summary. The teachers participating in IN-STEP are generally experienced and they all 
majored in science at the university. They tend to work in the larger schools in Phang-nga. The 
majority are from schools whose performance on the GAT science test is higher than the national 
average. For the most part they are participating willingly in IN-STEP because they hope to 
improve their teaching techniques. They tend to work in isolation and report low levels of 
collaboration with other teachers in their schools. However, they describe their principals as 
being supportive of science and their participation in the project. 

V. The IN-STEP Instructional Teams 

In the IN-STEP project, professional development for teachers is planned and delivered by an 
instructional team consisting of accomplished teachers and content experts. These instructional 
teams typically have four to seven members who plan the workshop for a single curriculum 
module and then deliver the workshop for a group of 15 teachers. Some of them also serve as 
mentors to provide follow-up support as the teachers use the new modules. The formation of 
effective instructional teams depends on careful recruitment, a well-designed process of team 
formation and workshop design, and the engagement of team members over time to provide 
continuity and experience. 

Recruitment. In Thailand, MISE provided staff with expertise in science and experience with 
the instructional team approach to lead the initial teams. K.I.Asia, with assistance from the 
ESAO and IPST, recruited accomplished Thai teachers and other science educators to serve on 
the teams. The initial teams were somewhat larger than those typically used by MISE as the level 
of interest in the process was quite high. 
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Table 16. Instructional Team Members for Cohorts 1 and 2 



Categories 


Year One 
Three Workshops 


Year Two 
Six Workshops 


MISE 


3 


5 


Active Teachers 


12 


24 


in Phang-nga 


Other Teachers 


3 


3 


University Faculty 


3 


0 


National Government 


6 


5 


Agency Staff 


ESAO Phang-nga 


3 


2 


Totals 


30 


39 



Source: IN-STEP database. 



Table 16 displays the background of the teams used to plan the first round of workshops (three 
curriculum modules/43 teachers) in March, 2007, and those who are expected to lead the second 
round of workshops (six curriculum modules/88 teachers) in April, 2008. It also is clear from 
Table 16 that the average size of the instructional teams is decreasing. This is intentional as the 
IN-STEP leaders found that it was awkward to have eight to 10 people on a team. There was 
simply not enough for them to do. In addition, the number of teachers engaged in the 
instructional teams is increasing, while the involvement of university and government personnel 
is decreasing. This has advantages and disadvantages. From the perspective of building capacity 
in the province to provide professional development in science and from the perspective of 
developing teacher leadership, it is a positive trend. But from the perspective of developing 
understanding of the model in national agencies and engaging the content expertise of university 
faculty, it is a negative trend. 



Design Retreats. The instructional teams prepare for their work in a three-day design retreat, 
and it is there that they begin their collaborative effort to design an effective teacher workshop 
for the curriculum module assigned to them. So far, the MISE staff has conducted two design 
retreats, one in October, 2006, and another in October, 2007. At the retreat the teams are 



26 




IN-STEP: Evaluation of the Eirst Year 



provided with expert assistance from experienced science educators and professional developers, 
attend sessions where big ideas in pedagogy, assessment, and learning are reviewed and 
discussed, and have large blocks of time to plan their own workshops. 

Team members are provided with the science module that they will be working on in advance so 
that they can teach all or part of it in their own classrooms, and familiarize themselves with the 
content, the investigations, the problems students encounter, and the assessments. At the retreat, 
they review the module as a team, identify areas that teachers and/or students might struggle 
with, and begin to map out a set of experiences to be offered to teachers that will introduce them 
to the module and give them first-hand experience with the investigations they will eventually be 
asking their students to conduct. After the retreat, the teams continue their work so that they have 
a well- specified script for their workshop long before the time comes to deliver it. 

Team members who are working on a module that was the subject of a workshop the previous 
year are charged with considering the feedback from the previous year's participants as well as 
the notes of the previous instructional team members to make revisions in the workshop script. In 
this manner, the workshops become more highly specified over time and hopefully more 
effective as they become more responsive to the needs of teachers. 

Table 17 displays the instructional team members’ assessments of the preparation they received 
in the design retreats. While they were asked a number of specific questions about trainers, time 
allocations, materials, their confidence about doing the work, etc., their responses were 
uniformly high. This reflects both the quality of the retreats and the enthusiasm of the 
participants. As a consequence, we have reported only the summary scores here. 
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Table 17. Instructional Team Members’ Ratings of Design Retreats 
Years 1 and 2 [Based on a Four- Point Likert Scale] 



Module 


Year 1 Teams 
N = 30 


Year 2 Teams 
N = 39 


Earth in Space 


- 


3.65 


Energy, Machines, 
and Motion I 


- 


3.92 


Light 


- 


3.94 


Properties of Matter 


- 


4.00 


Energy, Machines, 
and Motion II 


- 


3.64 


Human Body Systems 


- 


3.87 


Overall Average 


3.33 


3.84 



Note: In Year One, neither the participant name nor module name was collected. 

In Year Two, there was information about the module name. 

Continuity. Twenty-three of the 30 instructional team members involved in the first IN-STEP 
workshop — and 100% of the instructional team members who were from Phang-nga — 
voluntarily participated in the second round. This is important to the success of IN-STEP 
because it is this accumulated knowledge and skill that will make the program increasingly 
robust. It also is an indicator of the importance and quality of the program. These are busy 
people, and they are donating their time, so their commitment reflects their judgment that this is 
a worthwhile endeavor. 

The instructional team approach has been used for nearly two decades in the United States by 
MISE and has four enormous advantages over conventional training models: 

1 . The overall quality of the teams is quite high as they are not dependent on a single 

individual. Instructional team members possess both content and pedagogical knowledge, 
and can share both theoretical and practical perspectives on the use of the curriculum 
materials. The presence of content experts offers teachers opportunities to deepen their 
understanding of the content and to master different ways of representing concepts for 
students. The presence of accomplished teachers provides insights into how the lessons 
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will flow in the classroom, what problems are likely to occur, and how teachers can deal 
with the classroom management issues that arise when students are engaged in inquiry. 

2. The instructional teams provide rich opportunities for the development of teacher 
leadership in science. Teachers who serve on the teams deepen their content knowledge, 
make their tacit craft knowledge about pedagogy explicit, learn how to design effective 
learning opportunities for teachers, and become confident presenters and facilitators. 

3. The design retreat offers an opportunity to pull the team together as a working group and, 
after the first year, it provides an opportunity to apply the lessons learned from the 
previous year and improve the training experience for teachers. The design retreat is a 
critical piece of the MISE model as it provides professional development for the 
instructional teams themselves and strengthens their capacity to work with teachers. 

4. The instructional teams expand the pool of individuals qualified to lead professional 
development, providing the capacity for scaling up the program. Over two years, 34 
Thais have served on instructional teams. By the fourth year, the number of experienced 
professional development leaders is likely to reach 60. This means that for each of the 
nine curriculum units being implemented under IN-STEP, there will be six to eight 
experienced trainers who could organize new teams to train in other provinces. Based on 
the MISE experience, two to three experienced people can be matched with two to three 
new trainers to create an effective instructional team, so there will be the capacity to 
create three instructional teams for each unit. 

Summary. Using the instructional teams to design and deliver the IN-STEP workshops for 
teachers appears to be working as well in Thailand as it has in the United States. It has not been 
difficult to recruit qualified people to serve on the teams. Those recruited have been enthusiastic 
about the training that they have received. However, it is worrisome that the participation of 
university faculty and government agency staff has dropped off significantly. And, to date, the 
process has been led by the MISE staff. The expansion of IN-STEP over the next two years will 
require the recruitment of additional instructional team members and the development of local 
leadership for the teams. 
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VI. IN-STEP Professional Development 

In this section, we examine the professional development provided to teachers by IN-STEP. 
Previously, we have described how the teacher workshops are developed. Each one is organized 
to introduce teachers to a particular science module. Three of these workshops were offered to 
teachers in April, 2007, and 43 teachers participated. In April, 2008, an estimated 88 teachers 
will attend six workshops, each one preparing participants to teach a different science unit. How 
have the participants rated these learning experiences to date? 

Participant Ratings of IN-STEP Professional Development. As indicated in Table 18, the 
ratings from the teacher surveys conducted at the end of the workshops show that the participants 
thought the professional development was of very high quality. The participants were very 
enthusiastic about the training. One said: “It is completely new; the IN-STEP materials are very 
good, the best. I give 100% to IN-STEP, and the project will be better if it is expanded to other 
schools in other provinces.” Another expressed the sentiments of many, saying: “I think IN- 
STEP is the first training [I have had] in my life that I can use in the real classroom because the 
project provides me everything I need such as materials, handbooks.” (IN-STEP field notes, 
March, 2007). These responses were typical. 

Many of the participants felt that IN-STEP had made them more effective teachers and made 
teaching easier. Eor example, one said: 

The training has helped me a lot, it made me aware of the right direction instead of 
having to plan everything by myself. IN-STEP also saves my time since I have so many 
assignments, and it makes me confident about teaching. Students receive better learning 
than from traditional style in which they just memorize the content. (IN-STEP field notes, 
March, 2007) 

Another said: “IN-STEP has made me confident, I surely teach the right way and students are 
more motivated.” The teachers were particularly positive about their increased understanding of 
science. They made comments such as: “IN-STEP prepared me well in the content and in using 
both the teacher manual and the student manual, which help make learning activities easier.” 
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Table 18. Overall Quality of the Professional Development for Teachers 



Survey Topics 


Average Rating 


Accomplishment of the Objectives 


4.32 


The Quality of the Training Activities 


4.62 


Confidence in Capacity to Implement 


4.58 


Benefit to the Participant 


4.58 


Overall Average 


4.53 



Source: Questionnaire on Professional Development, March, 2007; N = 42, Eikert Scale 5. 



Table 19. Ratings of the Professional Development Six Months After Training 



Ratings of the Instructional Teams and 




Module 






IN-STEP Workshops on a 4-Point Rubric 


EIS 


EMM 


L 


Total 


The workshop provided an excellent understanding of the science 
content and inquiry and there was excellent support by the 
Instructional Team. 


4 


3 


6 


13 


The workshop provided a generally good understanding of the 
content and inquiry and the support by the Instructional Team was 
adequate. 


6 


7 


8 


21 


The workshop provided a fair understanding of the content and 
inquiry and/or the support provided by the Instructional Team was 
lacking in some ways. 


1 


4 


1 


6 


The workshop provided a weak understanding of the content, the 
Instructional Team seemed to be confused, and/or the support 
provided was generally inadequate. 


2 


1 


0 


3 


Total 


13 


15 


15 


43 



Source: Telephone Interviews, October, 2007. 



Table 20. Level of Confidence in Content Knowledge Six Months After Training 



Self Ratings by Teachers on a 4-Point Rubric 


EIS 


Module 

EMM 


L 


Total 


Highly confident about content knowledge 


6 


0 


2 


8 


Somewhat confident about content knowledge 


3 


7 


9 


19 


Some lack of confidence in content knowledge 


4 


7 


2 


13 


No confidence in content knowledge 


0 


1 


2 


3 


Total 


13 


15 


15 


43 



Source: Telephone Interviews, September, 2007. 
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Table 21. Level of Preparedness in Use of Inquiry Six Months After Training 



Module 



Rating on a 4-Point Rubric 


EIS 


EMM 


L 


Total 


Highly confident about use of inquiry 


10 


6 


5 


21 


Somewhat confident about use of inquiry 


2 


4 


8 


14 


Eack confidence about use of inquiry 


1 


4 


2 


7 


No confidence in use of inquiry 


0 


1 


0 


1 




13 


15 


15 


43 



Source: Telephone Interviews, September, 2007. 



However, the ratings in Tables 19, 20, and 21, drawn from interviews conducted with the same 
teachers six months later, show that their assessments of the quality of the professional 
development had changed somewhat as they faced the realities of using the new curriculum 
materials in their classrooms. This was most noticeable in the Energy, Machines, and Motion 
module, the subject matter of which teachers probably had the weakest content background and 
reported the lowest levels of confidence in their preparedness. (See Table 20). Only six of the 43 
teachers were physics majors, which might account for this falloff. 

Interestingly enough, while other data (see Table 14) show that the teachers anticipated that IN- 
STEP would help them improve their teaching methods and learn how to use inquiry in their 
classroom, and did not expect the workshops to deepen their knowledge of science, in the 
interviews conducted while they were implementing the new curriculum, they expressed greater 
confidence in their skills at using inquiry (see Table 21) than they did in their knowledge of the 
scientific content (see Table 20). Nearly half of the respondents indicated that they were highly 
confident that they could use the inquiry process in their classrooms, but only one-fifth of them 
expressed high confidence that they could teach the science concepts in the modules. 

Table 22. Improvements in Teaching 



Issue Number 

IN-STEP makes the teacher more confident in classroom 6 

Student learning has increased 6 

Teaching is easier when teacher uses IN-STEP I 

Decrease in preparation time I 

Source: Telephone Interviews, September, 2007. 
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In the interviews, teachers also were asked what improvements they had made in their teaching 
as a result of participating in IN-STEP. Table 22 displays the frequency with which various 
improvements were mentioned in the teachers’ responses. The most frequent responses were an 
increase in self-confidence and improved student understanding of the content. Note that only 14 
of the 43 teachers responded to this item. This reflects the inexperience of the IN-STEP 
evaluation team in that secondary questions meant to extend and elaborate responses, provide 
examples, or probe deeper into a topic were sometimes neglected. As indicated in Section III, we 
are working to improve the interview skills of the team and the interview process. This is another 
aspect of IN-STEP’ s capacity-building agenda in that evaluation team members are learning how 
to use qualitative methods and how to study implementation processes. 

Teachers also were asked how the IN-STEP professional development could be improved. Again 
they responded from the perspective of six months of experience during which time they had 
attempted to use the module that they had been prepared to use. Their suggestions are displayed 
in Table 23. The most frequent response by far was a desire for interaction with the IN-STEP 
team and with other teachers after the workshop. About a quarter of the teachers wanted forums 
where their problems could get discussed and solved and where ideas about use of the modules 
could be shared. This was a strong response from the Energy, Machines, and Motion teachers 
who also expressed the least satisfaction with their preparation and the lowest confidence in their 
content knowledge. 



Table 23. Teachers’ Suggestions for Improving Professional Development 



Suggestions 


EIS 


Module 

EMM 


L 


Total 


More interaction after the PD 


2 


6 


3 


11 


Better instructional material 


1 


1 


4 


6 


Better project management 


2 


0 


3 


5 


More time in the workshop 


2 


0 


3 


5 


Better instructional team 


2 


0 


0 


2 


Expand to other school/level/subject 


2 


0 


1 


3 


Knowledge on assessment 


1 


1 


1 


3 


Better manual 


0 


1 


1 


2 


No response 


3 


7 


4 


14 


Total 


15 


16 


20 


51 



Source: Telephone Interviews, September, 2007. 
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Summary. For the most part, teachers hold IN-STEP professional development in high regard, 
but after some experience using the materials, some have suggestions for improving the 
experience. The most common suggestion is for more follow-up support. It is also clear from the 
data that teachers’ assessment of their professional development experience varies across the 
modules. Some modules may require more support because of the complexity of the material, the 
demands of the investigations on classroom management, the science backgrounds of the 
teachers, or all of the above. 

VII. The Use of the IN-STEP Materials and Pedagogy 

In this section we look at two critical questions: Are teachers using the new instructional 
materials? And how are they using them? The level of use — the implementation — of the first IN- 
STEP modules provides us with an important leading indicator about the potential impact of IN- 
STEP on teaching and learning. If teachers do not use the materials, then obviously the entire 
program will have little impact although it could still affect teachers’ pedagogical practices. And 
if the materials are not used as they were designed to be used, the likelihood is that there will be 
less impact on student learning. So where are we with implementation of the instructional 
materials at this point in time? 

Three groups of teachers were trained in the use of one of three modules in March, 2007, but due 
to problems with shipping and customs, they did not receive the modules until August, 2007. 
Many had planned to begin teaching the modules in June and July. The late arrival of the 
materials meant that some of them had already begun other units and others were reluctant to 
begin to use the new modules for fear they would not be able to finish them before the semester 
ended in October, 2007. So there were further delays in implementation. These problems may 
have had significant impact on the level of use of the modules in this first year. 

To measure implementation, we asked the teachers to describe their use of the module in an 
interview conducted toward the end of the period in which they were using it. We specifically 
asked them if they had used, or planned to use, all of the lessons in the module and if they had 
made any adaptations to the lessons, and to tell us about the nature of those adaptations. We then 
coded these interviews and grouped teachers into the following categories: 
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• Implementing with high fidelity, meaning that they were using all of the lessons and 
made only minor adaptations (adding an inquiry, extending a lesson) 

• Partial implementation or implementing with major adaptations, such as making major 
changes in the sequence of lessons, skipping some investigations, using other materials in 
place of the IN-STEP material, etc. 

• Selective use of materials from the module as a supplement to the old curriculum 

• No implementation to date 

In Table 24 we see that only 7 of 43 teachers reported implementing the modules with high 
fidelity, meaning with no adaptations to or omissions from the original design. This is probably 
less fidelity than the IN-STEP team hoped for, but it is understandable given the context, the 
limited experience of the teachers with instructional materials of this kind, and the delayed 
arrival of the materials. A more positive interpretation of the data in Table 24 is that 30 of the 43 
teachers implemented the units with no or relatively small adaptations. Note the variation across 
the three units: Teachers using the Eight unit made the largest number of small adaptations as a 
result of the problems they had obtaining halogen bulbs, but also had the highest overall level of 
implementation. 



Table 24. Level of Implementation by Module 



Level of Implementation 


EIS 


Module 

EMM 


L 


Total 


High (fidelity) 


3 


3 


1 


7 


Moderate (adaptation or partial) 


3 


7 


13 


23 


Low (selective use as supplement or limited use) 


7 


5 


1 


13 


Total 


13 


15 


15 


43 



Source: IN-STEP database. 
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Table 25. List of Adaptations by Module (Number of Mentions) 



List of Adaptations 


EIS 


Module 

EMM 


L 


Total 


Teachers change method or materials 


3 


3 


6 


12 


Teachers switch the lessons or activities 


0 


0 


2 


2 


Teacher cut some lessons or activities 


4 


5 


3 


12 


Teachers extend time 


2 


2 


4 


8 


Teachers provide additional science contents 


1 


0 


2 


3 


Teachers provide additional guidelines 


0 


4 


5 


9 


Total 


10 


14 


22 


46 



Source: IN-STEP database. 



Table 25 provides us with some insights into the nature of the adaptations teachers made. 
Teachers using the Light unit made the most adaptations, largely because of the problems with 
the light bulbs, but also added new material and more time, and provided additional guidance to 
students. For the most part, these were small, nonlethal adaptations — that is, they did not alter 
the basic design, or the content, or flow of the lessons. 

Data provided by the mentors confirm the teacher self-reports on implementation. The data in 
Table 26, while not precisely parallel to the data in Table 24, show that nearly 90% of the 
teachers were using the material but that two-thirds of those had made some modifications to the 
units. The mentors also report that the highest number of modifications were in the Light unit. 
Encouragingly, the mentors report that 28 of the 31 teachers who reported using the modules said 
planned to use the material again the next semester. The IN-STEP team needs to probe why three 
teachers said they were not planning to use the units and, if their decisions cannot be justified or 
altered, consider whether they should be dropped from the program. 

Table 26. Mentors’ Reports on Observed Implementation 



Teachers' Level of Implementation 


EIS 


Module 

EMM 


L 


Total 


Number of teachers visited (mentees) 


9 


15 


12 


36 


Teachers using IN-STEP module 


6 


13 


12 


31 


Teachers planning to use IN-STEP again next semester 


6 


13 


9 


28 


Teachers making modifications to the IN-STEP module 


3 


8 


11 


22 



Source: Mentor Questionnaire October 2007. N = 14 (of 17); N of Teachers = 36 (of 43). 
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Problems With the Materials. Teachers reported a variety of problems with the IN-STEP 
materials (See Table 27). The problems varied across the modules. The teachers using the Light 
unit had trouble with the halogen bulbs and some of the investigations were dependent on having 
the bulbs. The teachers using the Energy, Machines, and Motion module reported more 
breakage. They also had more problems with the student and teacher manuals due to poor 
translations, the flow of the materials, and correlating the two manuals. 

Table 27. Difficulties With Use of the Units (Number of Mentions) 



Participation 


EIS 


Module 

EMM 


U 


Total 


Translation of the IN-STEP teacher manuals 


4 


12 


4 


20 


Translation of the IN-STEP student manuals 


1 


12 


4 


17 


Understanding the processes in the manuals 


7 


2 


4 


13 


Missing or broken materials 


3 


9 


15 


27 


Students’ lack of confidence 


1 


1 


3 


5 


Students’ lack of science background 


1 


1 


0 


2 


Needed to more time 


0 


3 


2 


5 


Teachers’ lack of assessment method 


0 


1 


0 


1 


Teachers located too far from mentors 


0 


1 


0 


1 


No problems reported 


4 


3 


0 


7 


No response 


2 


1 


0 


3 


Total 


23 


46 


32 


101 



Source: Telephone Interviews, September, 2007. 



Student Response to the Materials. If students respond enthusiastically to new materials, and if 
they demonstrate success on the assigned tasks and on teacher assessments, then teachers are 
more likely to continue using the material. How have students responded so far to the IN-STEP 
materials? The only data we have come from teacher self-reports, which might be inclined to be 
somewhat positively biased. Nevertheless, the data presented in Tables 28 and 29 are 
encouraging as they show that student engagement in the materials was generally good and for 
the most part students were able to master the content. However, a significant number of 
teachers — almost 20% — reported that student engagement was low. This may reflect the way 
the materials were introduced, managed, and used by the teacher. This is an area that requires on- 
site investigation. The data in Tables 28 and 29 also suggest that student engagement and 
mastery of the material were lower for the Light module than for the other two modules (Earth in 
Space and Energy, Machines, and Motion). Whether that has to do with the equipment problems 
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that teachers experienced, the level of expectations of the teachers, the difficulty of the material, 
or the way the material was taught is not clear. Here again, there is a need for direct observation 
and more targeted interviews of the teachers. 

Table 28. Student Engagement 



Participation 


EIS 


Module 

EMM 


L 


Total 


High 


4 


3 


1 


8 


Medium 


5 


8 


7 


22 


Low 


0 


1 


6 


7 


No response 


4 


3 


1 


7 


Total 


13 


15 


15 


43 



Source: Telephone Interviews, September, 2007. 



Table 29. Student Performance on Teacher Assessments 



Level of Mastery 


EIS 


Module 

EMM 


L 


Total 


High 


4 


3 


1 


8 


Medium 


6 


8 


7 


21 


Low 


0 


1 


6 


7 


No Response 


3 


3 


1 


7 


Total 


13 


15 


15 


43 



Source: Telephone Interviews, September, 2007. 



Factors Affecting Implementation. The research literature identifies a number of factors that 
affect teacher adoption of new practices. Most of this literature is based on studies in western 
countries, and often biased toward the experiences of urban schools. Among the critical variables 
are school size, principal support for the reforms, the level of professional collaboration in a 
school, the age and experience of the teachers involved, and the willingness of teachers to adopt 
the reforms. Here we look at how these factors appear to be affecting the implementation of the 
IN-STEP materials. 

Table 30 displays the cross-tabulation between school enrollment and the level of 
implementation. The data show a statistically significant relationship between size and the level 
of implementation with teachers in larger schools generally implementing the modules with 



38 




IN-STEP: Evaluation of the Eirst Year 



greater fidelity. Fisher’s Exact Test can be interpreted like a chi-square test, but it does not 
require that the Ns in 80% of the cells be greater than 5. So in this table, the likelihood that there 
is a relationship between the two variables — school size and level of implementation — is greater 
than 99%. 



Table 30. School Size and Level of Implementation 



Level of 
Implementation 


Small 


Medium 


Large 


Totals 


High fidelity 


1 


3 


3 


7 


Moderate adaptation 
or partial 


2 


13 


8 


23 


Low (use as 
supplement) 


1 


0 


12 


13 


Totals 


4 


16 


23 


43 



Note: Eisher’s Exact Test p. = 0.0018. Highly significant. 

In Table 31, we examine the relationship between each school’s average performance on the 
GAT science test and implementation and find no relationship between the two variables. Given 
that the GAT is a fact-oriented test, this perhaps should not be too surprising. In Table 32, the 
relationship between teachers’ perceptions of their principals’ support for IN-STEP and 
implementation is examined, and here we also find a strong relationship. Teachers are more 
likely to implement the full units if they believe that their principals are supporting their effort. 

In western societies, this relationship is typically quite high in elementary schools where 
principals are expected to be instructional leaders, but weaker at the secondary levels where 
teachers are content specialists and principals are managers. As the amount of time devoted to 
IN-STEP materials increases and if the level of student engagement in inquiry increases, we 
might expect to see principals even more involved in decisions about the sustained use of the IN- 
STEP materials and see an even stronger relationship here. The increased use of IN-STEP 
materials will generate some pressure on instructional materials budgets as consumables have to 
be replaced, and some pushback from principals might be anticipated as they struggle to deal 
with competing demands. And students will become more active learners, and perhaps noisier, 
also drawing the attention of some principals. On the positive side, there will be more student 
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work product to examine and perhaps more students involved in science fairs and science 
competitions. These developments also will draw the attention of principals. 

Table 31. Average Performance on the GAT Science Test 
by School and Level of Implementation 



(N = 42) 


Level of Implementation 


Below Average Achievement 


Above Average Achievement 


High fidelity 


2 


5 


Moderate adaptation or partial 
implementation 


12 


11 


Low (use as supplement) 


5 


8 



Note: Chi-square = 0.906, p = 0.6357. Not significant. 



Table 32. Principal Support and Level of Implementation 



Level of 
Implementation 


Unsupportive 
Or Neutral 


Supportive 


Totals 


No 

Response 


High fidelity 


1 


5 


6 


1 


Moderate adaptation or 


8 


15 


23 


0 


partial 










Low (use as supplement) 


12 


1 


13 


0 


Totals 


21 


21 


42 


1 



Note: Eisher’s Exact Test p = 0.000418. Highly significant. 

In Table 33 we look at the age of the teachers and their level of implementation and find a weak 
inverse correlation. The data suggest that younger teachers were somewhat more likely to 
implement the full units, but show little variation among teachers over the age of 30. 
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Table 33. Teacher Age and Level of Implementation 



Level of 
Implementation 


Under 30 


31-40 


41-50 


Over 

50 


Totals 


High fidelity 


3 


1 


2 


1 


7 


Moderate adaptation or 
partial 


3 


9 


3 


8 


23 


Low (use as supplement) 


1 


5 


6 


1 


13 


Totals 


7 


15 


11 


10 


43 



Note: Eisher’s Exact Test p = 0.1430. Not significant. 



Table 34. Teachers’ Decision to Participate and Level of Implementation 



Level of 
Implementation 


Voluntary 


Assigned 
but Willing 


Assigned & 
Reluctant 


Totals 


High fidelity 


5 


2 


0 


7 


Moderate adaptation or 
partial 


8 


11 


4 


23 


Low (use as supplement) 


6 


7 


0 


13 


Totals 


14 


7 


21 


42 



Note: Eisher’s Exact Test p. = 0.3358. Not significant. 

In Table 34 we find no statistieal relationship between voluntary partieipation and the level of 
implementation. This might be due to the eonfounding of two eategories — voluntary and 
assigned but willing; that is, many individuals in the seeond eategory might have attended 
voluntarily if they had the opportunity to do so. 
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Table 35. School Penetration and Level of Implementation 



Level of 
Implementation 


Less 

Than 

Half 


More than 
Half 


All 


Totals 


High (fidelity) 


3 


2 


2 


7 


Moderate (adaptation 


7 


14 


2 


23 


or partial) 


Low (selective use as 


5 


5 


3 


13 


supplement) 


Totals 


15 


21 


7 


43 



Note: Eisher’s Exact Test p. = 0.7506. Not significant. 

In Table 35, we display data on the relationship between the level of implementation and the 
proportion of a school’s faculty who participated in IN-STEP in the first year. We refer to this as 
the degree of penetration. The hypothesis is that implementation is likely to be higher if the 
teacher has colleagues who are working to make the same or similar changes in their practice. 
Here again, we find no relationship at this point. This is surprising, but it might be because in the 
instances where there are multiple teachers from the same school participating in IN-STEP, they 
are teaching different modules at different grade levels and therefore are not well positioned to 
help one another. 

Table 36 examines yet another hypothesis drawn from research in schools in western nations — 
that the level of teacher collaboration in a school or department is positively related to the level 
of implementation of instructional reforms. But as we saw earlier, the level of collaboration in 
the schools in Phang-nga is quite low, so teachers may be unaccustomed to this kind of support 
and, therefore, collaboration may have little bearing on their decisions to use or not use new 
instructional materials. However, the data do show a weak positive relationship between reported 
levels of collaboration and the level of implementation. 
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Table 36. Level of School Collaboration and Level of Implementation 

(N = 42) 



Level of 
Implementation 


None or Weak 


Moderate 


High 


Totals 


High (fidelity) 


3 


2 


2 


7 


Moderate 
(adaptation or 
partial) 


18 


3 


1 


22 


Low 

(selective use as 
supplement) 


6 


5 


2 


13 


Totals 


27 


10 


5 


42 



Note: Eisher’s Exact Test p. = 0.0865. Not significant. 



Summary. Overall, the rate and level of implementation of the first units were about what one 
would have expected, perhaps better than might have been expected given the problems with 
delivering materials, the time of the year, and the degree of classroom change being asked of 
teachers. Nevertheless, there was a lot of adaptation and a number of cases of weak 
implementation and use of the materials as supplements rather than the primary instructional 
medium. All of this suggests that clarifying expectations for participants, giving more attention 
to anticipated problems at the workshops, providing more follow-up support (perhaps through 
regional teacher meetings), and briefing principals are needed to improve the level of 
implementation in the second round. 

VIII. The Mentors 

The IN-STEP mentors are accomplished teachers who possess knowledge about content, 
pedagogy, assessment, standards, and curriculum and have volunteered to assist teachers with the 
implementation of the IN-STEP units and improving their instructional skills. They have been 
trained in classroom observation and other skills needed to assist teachers. They seek to build 
trusting relationships with teachers so that they are welcome in classrooms and can provide 
effective feedback and engage in reflective conversations about teaching practices. Through such 
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constructive meetings, mentors are able to assist teachers with planning, teaching, assessing, and 
reteaching to improve student performance. They collect information about students, teachers, 
classrooms, and schools and keep it confidential. They share the information only with the IN- 
STEP research team. 

The Recruitment of the Mentors. Mentors were recruited from the IN- STEP instructional 
teams on a voluntary basis. They were enthusiastic in taking on this mentorship role, motivated 
as they were by the opportunity to develop their professional skills and to support their teacher 
peers. In the first year, they attended training in June, 2007, and October, 2007. The first session 
focused on developing observation and meeting facilitation skills whereas the second training 
focused on developing questioning and feedback delivery skills 



Table 37. Perceptions of the Quality of the Mentor Training 



Overall Mentor Workshop Quality 


Training Session 




Module 






EIS 


EMM 


L 


Overall 


Session 1 (4 pt scale) 


- 


- 


- 


3.68 


Session 2 (5 pt scale) 


4.42 


4.64 


4.92 


4.6 



Source: Session 1 Mentor Questionnaire June, 2007, N = 17; 

Session 2 Mentor Questionnaire October, 2007, N = 14. 



Table 37 shows that the mentors rated the first training session highly, with an average rating of 
3.68, based on a four-point scale. In the second training, a five-point scale was used and the 
average rating was about 4.6. 

Activities of Mentors. Each mentor was assigned to work with two to three teachers whose 
schools were in close proximity to the mentor's location. They were asked to organize a meeting 
with the teachers assigned to them to discuss the use of IN-STEP and related pedagogical issues, 
and then to meet with each teacher individually. Thirteen of 14 mentors organized the two 
meetings. 
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Table 38. Conducting Teacher Meetings 



Did mentor meet teacher? 


EIS 


Module 

EMM 


L 


Total 


Yes 


4 


5 


4 


13 


No 


1 


0 


0 


1 


Total 


5 


5 


4 


14 



Source: Mentor Questionnaire October, 2007, N = 14 (of 17). 



Table 39. Attendance at Teacher Meetings 



Did all teachers 
attend the meeting? 


EIS 


Module 

EMM 


L 


Total 


Yes 


3 


5 


4 


12 


No 


2 


0 


0 


2 


Total 


5 


5 


4 


14 



Source: Mentor Questionnaire October, 2007, N = 14 (of 17). 



Table 40. Teachers' Perception of Mentor Support 
(Four-Point Rubric) 



Scale 


Level of 
Support 


Quality of 
Interaction 


Level of Teacher 
Receptivity 


EIS 


2.53 


2.69 


3.76 


E 


3.00 


3.06 


4.00 


EMM 


2.93 


2.73 


3.60 



Source: Teacher Interviews, October, 2007. 



The data presented in Table 40 show that teacher receptivity to the mentors is uniformly high 
among the three groups of teachers, but in spite of that, the perceived quality of interactions 
with the mentors and the level of support provided were not regarded as highly by the 
teachers. The teachers using the EIS module were especially critical, rating the level of 
support and the quality of interaction much lower than teachers using the other two modules. 
These data are in agreement with those from Table 38 showing that some mentors did not 
organize the suggested teacher meetings. These data imply that there should be more 
interaction and stronger support mechanisms between mentors and mentees. The IN-STEP 
team should explore how mentors can provide better support and what additional training is 
needed for them to carry out the mentorship task more effectively. 
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Table 41. Benefits of Mentoring to Teachers 



Improvement 


EIS 


Module 

EMM 


L 


Total 


Teachers' confidence in science 


1 


3 


0 


3 


Teaching techniques on inquiry 


2 


2 


3 


7 


Confidence in using materials 


0 


3 


2 


5 


Classroom management 


1 


2 


1 


4 


No response 


1 


0 


1 


2 


Total 


5 


10 


7 


22 



Source: Mentor Questionnaire October, 2007, N = 14 (of 17), Double Count. 



Based on 14 of 17 mentors, Table 41 shows that mentors believe that they can help teachers 
gain confidence in their knowledge of science, develop their skills at using inquiry, acquire 
confidence in using the material, and improve classroom management. It is noteworthy that 
they reported that teachers in EIS and Light did not improve much in their content 
knowledge. In addition, teachers in Earth in Space were perceived as not improving their 
confidence in using material. These teachers were less likely to attend the teacher meetings 
organized by the mentors. It will be interesting to compare the similarities and differences of 
the perceived benefits of the mentorship to teachers against teachers’ reports of what they 
gained from their mentors. It will be useful to explore whether the problems in the 
relationships come from the mentors’ side or the teachers’, whether mentors need additional 
training in those areas, or whether teachers need training in how to work with a mentor. 
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Table 42. Benefits to the Mentors (Number of Mentions) 



Improvement 


EIS 


Module 

EMM 


L 


Total 


Knowledge 


0 


0 


1 


1 


Teaching technique 


1 


3 


0 


4 


Mentor technique 


1 


1 


1 


3 


Networking 


3 


3 


1 


7 


Experience 


0 


2 


0 


2 


Questioning skills 


1 


3 


1 


5 


Using materials 


1 


0 


0 


1 


No response 


1 


0 


2 


3 


Total 


8 


12 


6 


26 



Source: Mentor Questionnaire October, 2007, N = 14 (of 17), Double Count. 



The data in Table 42 show a mixed pattern of perceived benefits to the mentors themselves. It is 
noteworthy that the mentors do not believe that they have gained in content knowledge. 
However, five of 14 mentors reported that they improved their questioning skills, an area in 
which they received explicit training in October, 2007. The team needs to consider what 
knowledge and skills are needed to improve the efficacy of the mentoring process and then 
develop a structured training plan for mentors over the next two years. 

Summary. The initial feedback about the mentoring process shows that it has worked rather 
well. The mentors have carried out the responsibilities assigned to them and report that teachers 
are receptive to their support. The feedback from the teachers is more mixed; they do not all 
perceive the mentoring to be highly useful. However, the majority are positive and the process 
can be expected to improve. 



IX. The Impact of IN-STEP to Date 

What has been the impact of IN-STEP to date? At the national level, there is continued interest in 
IN-STEP. In fact, if imitation is the highest form of compliment, then IN-STEP was 
complimented when the IPST adopted some key aspects of the model for the design of its new 
science initiative in 10 other provinces. Eike IN-STEP, this initiative is providing intensive, 
curriculum-based professional development to teachers, but it does not provide them with 
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instructional materials and it relies on university faculty to provide the training. At the provincial 
level, the Phang-nga ESAO is highly supportive of IN-STEP and has opened a resouree center to 
replenish the modules. 

At the school level, it is too soon to tell. Two leading indicators might be how many of the 
eurrent participants return for the second round of workshops and how many teachers volunteer 
to participate in the next cohort from the schools that have already sent teachers to the 
workshops. If almost all of Cohort 1 participate in the second round of workshops, this would be 
a positive indicator about the likelihood of continued use of the modules, and if many additional 
teachers volunteer from the schools now participating, this could be interpreted as a positive sign 
that IN-STEP is attracting the attention of teachers and taking root in these schools. Conversely, 
if few volunteer, this would be a discouraging sign about future implementation beeause it would 
indicate that those teaehers who have been exposed to IN-STEP are not being excited by it, or 
are put off for some reason. 

At the classroom level, it is also simply too soon to say. We know that almost all of the teachers 
used the new currieulum materials, but we also know that only seven of 43 used them exactly as 
they were designed. And we know there are powerful forces — negative teachers and principals, 
pressure to prepare students for exams, and new admissions tests in some of the province's high 
schools — that could undermine the use of the IN-STEP curriculum materials. In the future we 
will seek better measures of classroom use of the modules and also seek to link student mastery 
of the concepts in the modules to the level of use of the IN-STEP materials. 

So IN-STEP is already having some impact, but it is too soon to assess its impacts on teaching 
practice and student learning, and these are the critical outcomes. 

One major challenge in assessing impact on student learning is obtaining standard measures of 
student performance from the schools. The new National Assessment samples schools and 
therefore is probably not useful for evaluation of the project. IPST has offered to create an 
assessment for 9* graders that is sensitive to the content of the nine IN-STEP modules, but this 
assessment does not yet exist and it remains to be seen if teachers and students will take it 
seriously. However, it may be the only way to obtain a good measure of the pilot’s impact on 
science learning. 
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X. General Findings, Challenges, and Recommendations 

Here are the key findings from the evaluation of IN-STEP after two years of project development 
and one year of implementation: 

1. Enthusiasm for the instructional reforms and for the IN-STEP approach to reform 
remains high among the partners, including provincial and IPST officials, principals, and 
teachers. People feel the strategy is thoughtful and likely to work. The teachers who have 
attended have for the most part attended willingly and with enthusiasm. Teachers 
attending the workshops report that most of their colleagues who have not yet attended 
feel positive about IN- STEP. 

2. Teachers participating in IN-STEP have been recruited primarily from the larger schools 
in the province. Many of the larger schools sent two or more teachers. Twenty-four small 
schools have yet to send teachers. This has implications for future recruitment and for the 
provision of implementation support (see below). 

3. Teachers, instructional team members, and mentors have a generally high opinion of the 
training they have received to date and feel well-prepared for the work they are expected 
to do. Teachers are the most critical group, but the most common criticism is that they 
need more of the kind of training they are receiving. 

4. The level of use of the IN-STEP instructional materials is relatively high — 36 out of 43 
teachers actually used all or most of the curriculum modules they received. However, 
they made many adaptations and implementation was uneven, in part due to the late 
arrival of the materials (near the end of the semester) and in part due to reluctance on the 
part of teachers to totally give up the familiar old curriculum. 

5. The only information we have about the response of students comes from their teachers. 
We might expect that data to be biased to the positive side since teachers would likely 
feel that negative responses would reflect on their teaching and would reflect poorly on 
their students. Keeping that potential bias in mind, the findings are mixed. About 80% of 
the teachers report that their students are moderately to highly engaged by the materials, a 
less enthusiastic response than might have been expected. But inquiry is new to the 
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students as well as the teachers and demands more of them, so perhaps this is a 
predictable response. The same percent of teachers report that their students are able to 
master the material. This leaves a significant group, about 20% of the teachers, who are 
reporting problems with engagement and mastery. This group disproportionately comes 
from the Light unit. These teachers need to be targeted for more support from the mentors 
in the next round, and the Light instructional team needs to explore the nature of these 
problems and address them in the next workshop. 

6. The support structures put in place by IN-STEP are functioning. Almost all of the 
teachers were visited by a mentor but only two-thirds of those visited felt this support 
was useful. While it is too soon to tell if the planned supports are adequate, they might 
not be robust enough to deal with the teachers from the many small schools in the 
province that have yet to send teachers to participate. 

7. Teachers report very low levels of collaboration in their schools. When asked who they 
turn to for support in science, half say they rely on themselves. There is little evidence of 
functioning professional communities in science in the schools or across schools in the 
province. The good news is that IN-STEP has an opportunity to improve conditions for 
science teaching. The bad news is that it seems unlikely that peer support will be robust 
enough in many schools to ensure implementation. Hence, we have concluded that in 
addition to on-site mentoring, periodic network meetings should be organized in several 
regions of the province to provide implementation assistance and model collegial support. 

8. Almost all of the volunteer mentors made efforts to provide support to the teachers 
assigned to them. And the teachers indicated that they wanted support form the mentors. 
However, the teachers gave mixed grades to the support they received from the mentors. 

Challenges and Recommendations. Seven major challenges face the IN-STEP team in the next 
year: Getting teachers to implement the modules as they were designed at least for their first use 
of the materials; building more collaborative cultures among teachers in the province; targeting 
recruitment this year and preparing to support teachers working in isolation in small schools by 
strengthening the overall quality and quantity of implementation supports; helping teachers get 
students more engaged in inquiry; expanding the instructional teams and developing local team 
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leadership; managing the resource center and effectively replenishing and redistributing the 
modules; and collecting data on student performance in science. 

Implementing with Fidelity. Only seven of the 43 teachers implemented the first modules 
exactly as they were designed. And at least 13 teachers did not implement the full module during 
2007, using only selected materials to supplement their old text. This suggests three possible 
problems: (1) The modules make new demands on teachers in terms of time, methods, or 
equipment that cannot be met; (2) the preparation and support provided for the teachers are not 
adequate to the demands of implementation; or (3) the expectations that the units be fully 
implemented have not been made clear enough. The available data do not provide a clear answer 
and the best guess is that all three of these problems are affecting implementation to some 
degree. The evaluation team will probe the causes of incomplete or partial implementation in the 
coming months. In the meantime, it is recommended that expectations for full and faithful 
implementation be made clear to both principals and participating teachers. This means that IN- 
STEP teams will need to make sure that the principals understand that the recruited teachers 
must be scheduled to teach a regular class that uses the module they are being trained to teach. 
Second, it is recommended that teachers who did not implement the module in 2007 not be 
invited to continue in IN-STEP unless they commit to implementing both the first and second 
modules in 2008. Third, it is recommended that the instructional teams stress fidelity on first use 
of the module and that they include a trouble-shooting session in the workshop to address 
common problems that may arise. 

Building More Collaborative Professional Cultures. The data show that collaboration is not 
the norm in the schools. Moreover, the current implementation support from mentors is being 
given mixed reviews from the teachers. It seems unrealistic to develop collaborative school 
cultures in the short run as the combination of professional norms, experience, and 
organizational conditions mitigates against it. However, the support of peers and peer pressure 
are important factors in implementing new instructional approaches and changing practice. How 
then to resolve this dilemma? We recommend that IN-STEP provide teachers with experience in 
collaboration by conducting periodic teacher meetings led by mentors and IN-STEP staff. These 
might be organized for three or four clusters of schools and occur three times a year. The 
meetings could be forums for teachers to share their experiences with the new content and 
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instructional materials, solve common problems, introduce new ideas such as formative 
assessment, or strengthen critical skills such as questioning strategies. 

Focusing Recruitment Efforts. Given the conditions in the schools, the need to strengthen 
support structures, and the increased engagement of teachers from smaller schools who work in 
isolation and who were not early volunteers for the program, the IN-STEP team needs to think 
carefully about its recruitment and roll-out strategies. One strategy would be to focus on scaling 
up participation of faculty in the schools that are already involved in the project as much as 
possible, perhaps limiting the number of new sites that are brought into the project each year. 
The advantages of this strategy are twofold: First, it might increase both peer pressure and peer 
support in the schools, producing higher rates of use of the curriculum; second, it makes the 
management of the IN-STEP support system easier in that there will be multiple teachers in each 
school. The alternative strategy would be to intentionally recruit volunteers from the 31 schools 
that have not yet sent teachers to IN-STEP training, but to do it by clusters in order to build a 
critical mass of teachers engaged in each of three regions. The IN-STEP support system then 
might focus on teacher meetings in the clusters rather than school-level support. We favor the 
former strategy because we believe, from past experience and research evidence, that it is likely 
to have more positive effects on the level of use of the materials and changes in teacher practice. 
The two strategies might be used in sequence, focusing on rollout within schools in 2008 and a 
cluster approach to recruitment in 2009. 

Engaging Students in Inquiry. Inquiry is not only new for teachers, it is new for students as 
well. It makes new demands on their work effort and requires different dispositions and skills 
than those required by the more passive, teacher-centered instruction they are accustomed to. 
Thai students are not used to learning through questioning, critical thinking, analyzing, 
presenting and providing conclusions on their own. This is not an easy problem to solve but it 
cannot be ignored as poor student response will discourage teachers and in the end defeat the 
purpose of the reforms. Therefore, we recommend that IN-STEP consider developing some 
model lessons that help introduce students to inquiry and develop the skills they need to engage 
in this kind of learning. The successful British program. Thinking Science, might be used as a 
source of lessons that could be adapted for use in Thai classrooms. 



52 




IN-STEP: Evaluation of the Eirst Year 



Expanding the Instructional Team Capacity. As IN-STEP expands to offer training on nine 
modules and serve over 130 lower secondary science teachers in Phang-nga, the demand for 
instructional team members will increase. In the third year of the project, 36 to 45 educators must 
be recruited to serve on the instructional teams. For the most part, these new members are likely 
to be recruited from the ranks of the participating teachers, but it would be useful in several 
respects to recruit team members from the local universities. The combination of university 
faculty and accomplished teachers would provide expertise in both content and pedagogy. It is 
essential that all members of the instructional teams be familiar with the content of the modules 
and have some experience using the modules. 

Many instructional team members — 17 in 2007 — also serve as mentors, providing on-site 
support to teachers. They are donating their time to assist colleagues with the improvement of 
instruction. Again, as the project expands, more mentors will be needed and the demands on 
mentors will increase as IN-STEP begins to recruit in smaller schools where science teachers 
may work alone and to recruit teachers who are likely to be reluctant participants. 

Finally, the plan is to hand off the leadership of instructional teams to Thai science educators. 
One of the six instructional teams offering workshops in April, 2008, will be led by a Thai 
educator. This process will need to be accelerated to provide Thais with experience leading 
teams for all nine modules. 

Creating an Effective Resource Center. The Kenan Institute Asia and the ESAO have 
established a resource center to replenish the modules to ensure their continued use. K.I.Asia has 
provided a staff person to do this work and the ESAO has provided the building (and renovated 
it). This process will become more important, and more demanding, as the number of teachers 
using the modules and the number of modules in use both increase. To protect the investment in 
the modules and ensure their effective use, this operation will have to be well-managed. It will be 
important to examine how it works in this first round, and then make projections for what will be 
needed to handle the increased workload in 2009, 2010, and beyond. 
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Developing Student Performance Data. Finally, in order to demonstrate the impact of the new 
instructional materials and pedagogy on student learning, measures are needed that are sensitive 
both to the content of the modules and to students’ understanding and capacity to use inquiry. 
The current assessments do not meet this need. IPST has offered to develop a science assessment 
for Phang-nga that focuses on the content of the IN-STEP units and assesses the use of inquiry. 
Development of this assessment should be a priority for 2008 as the last opportunity to capture 
"baseline" data. So far, students have been exposed to only one module, but in October, 2008, 
they will experience a second one and we will no longer be able to capture the baseline data 
needed to measure the impact of IN-STEP on student learning. 
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Appendix A. The Context for Reform of Science Education in Thailand 

by Kessara Amomvuthivorn 

Thailand faces stiff economic competition from its neighbors in Southeast Asia. While 
Thailand’s economic growth was hailed as an Asian Miracle in the early 1990s, it has slowed in 
recent years in the face of increasing economic competition from rapidly growing countries in 
the region like China and Vietnam. Several factors are contributing to Thailand’s loss of 
competitiveness relative to neighboring countries in the region. First, Thailand’s scientific and 
technological (S&T) capabilities have been weakening since 1998. It ranked 32"‘^ in the world in 
1997 in both technological and scientific infrastructures but dropped to 48* and 53* place 
respectively by 2006. 



Table 1. Ranking of Scientific and Technological Capability, Thailand 



Infrastructure 


1997 


1998 


1999 


2000 


2001 


2002 


2003 


2004 


2005 


2006 


Technological 


32 


43 


47 


47 


48 


43 


20 


45 


45 


48 


Infrastructure 






















Scientific 


32 


43 


48 


47 


49 


46 


26 


55 


56 


53 


Infrastructure 






















Numbers of 


47 


47 


47 


47 


49 


49 


30 


60 


60 


61 



Countries 

Assessed 

Source: IMD, various years. 

One factor contributing to the decline in Thailand’s standing in scientific and technological 
capabilities is the low percentage of jobs in its major industries requiring such skills. From 
Figure 1, it is apparent that more than 90% of the workforce in Thailand’s main industries are 
employed in jobs that do not require S&T skills. Only in the petrochemieal industry is a majority 
of the workers — nearly 60% — required to have S&T skills. In fact, the International Institute for 
Management Development (IMD) ranked Thailand's labor market 37th out of 61 countries in 
terms of the availability of skilled labor (2006). 
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Figure 1 shows that most sectors in Thailand’s economy do not require large numbers of workers 
with technical and scientific skills. But, the educational level and skills of the available 
workforce in Thailand may contribute to this pattern of development since industries requiring 
workers with technical and scientific skills are more likely to be attracted to nations whose 
workers have these skills. The World Bank's Private Investment Climate Survey of Thailand 
found that the skill deficiencies of the available workforce were the number-one concern of 
companies operating in Thailand (2004). 

Figure 1. Percent of S&T and Non S&T Workforce Classified by Industry 




Source: Thailand Research and Development Institute (2004). 

While Thailand has received deserved praise for its successful efforts to expand access to 
education and raise the level of educational attainment as measured by years of schooling, its 
schools are not yet providing the quality of education needed to support sustained economic 
growth. If we examine how well Thai students have performed on international mathematics and 
science tests, we can see evidence of the quality problem. Results from the TIMSS and PISA 
assessments suggest that Thai secondary education students perform well below the average of 
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students in other participating nations and, most importantly, lag behind students in other East 
Asian countries. 



Table 2. Trends in International Mathematics and Science Performance 







1995 




1999 


Country 


Math 


Science 


Math 


Science 


Singapore 


608.59 


580.35 


604.39 


567.89 


South Korea 


580.72 


545.78 


587.15 


548.64 


Taiwan 






585.12 


569.08 


Hong Kong 


568.89 


509.73 


582.06 


529.55 


Japan 


581.07 


554.47 


578.60 


549.65 


Malaysia 






519.26 


492.43 


Thailand 


516.22 


510.04 


467.38 


482.31 


Indonesia 






403.07 


435.37 


Philippines 






344.91 


345.23 



Source: Third International Mathematic and Science Study (TIMSS), as cited in the World Bank (2005). 

In addition, results of the 2006 Program for International Student Assessment (PISA) science 
assessment indicate that 1 5 -year-old Thai students score well below their peers in other Asian 
countries in mathematics and science except for those in Indonesia. As Table 3 displays, Thai 
students on average scored more than 100 points lower in mathematics and science than students 
in four advanced Asian economies. As 74.7 points on the PISA scale represents one proficiency 
level (there are six levels on the assessment), this means Thai students are on average scoring 
more than one level lower than students in these other Asian nations. 
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Table 3. Program for International Student Assessment (PISA) 
Science Scores for Asian Nations, 2006 



Country 


Mathematics 


Reading 


Science 


South Korea 


547 


522 


556 


Taiwan 


549 


532 


496 


Hong Kong 


547 


542 


536 


Japan 


523 


531 


498 


Australia 


520 


527 


513 


Thailand 


417 


421 


417 


Indonesia 


391 


393 


393 


OECD Average 


498 


500 


492 



Source: OECD, PISA 2006 Science Competencies for Tomorrow's World. 



Even more worrisome, few 15-year-old Thai students scored at the highest levels on the 
mathematics and science assessments. Table 4 compares the percent of Thai students scoring at 
the two highest levels on the seven-point scale used by PISA with results from other Asian 
nations. These data reveal there are many fewer highly qualified students in the pipeline in 
Thailand than in the school systems of some of the nation’s economic competitors. 

Table 4. PISA Percent of Students Scoring in Categories 5 and 6 



in Science; All Participating Asian Nations 



Korea 


9.2 


1. 1 


10.3 


Taiwan 


12.9 


1.7 


14.6 


Hong Kong 


13.9 


2.1 


16.0 


Japan 


12.4 


2.6 


15.0 


Australia 


II. 8 


2.8 


14.6 


Thailand 


0.4 


0.0 


0.4 


Indonesia 


0.0 


0.0 


0.0 


OECD Average 


7.7 


1.3 


9.0 



Source: OECD, PISA 2006 Science Competencies for Tomorrow's World. 
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The data from TIMSS and PISA reveal a large achievement gap between Thailand and its 
economic competitors in the region. These indicators make it clear that it is critical to the 
nation’s economic development that action be taken to improve the quality of education in 
science and mathematics. 
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Appendix B. IN-STEP Curriculum Modules 

Each module is organized around a major topic in science and contains 4-8 weeks of lessons that 
are designed to build student understanding and include investigations that provide opportunities 
for students to do science. The modules contain all of the equipment and materials that are 
needed to teach the lessons and conduct the investigations. Developed in the United States, the 
modules have been translated and adapted for use in Thailand and serve as the foundation of the 
IN-STEP science education initiative. 

2007 

Eight 

Earth in Space 

Energy, Machine, and Motion I 

2008 

Properties of Matter 
The Human Body System 
Energy, Machine, and Motion II 

2009 

To be determined 
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Appendix C: Map of Phang-nga Showing the Location of Schools 

Within the Province. 
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Wanna 



Kapong 

Pittayakom 



Kapong 

Pittayakom 



Kapong 

Pittayakom 



Deebuk 

Phang-nga 



Deebuk 

Phang-nga 



Suda/ 

Manas 

Suthapom/ 

Anonopatra 



Deebuk 

Phang-nga 



Deebuk 

Phang-nga 



Anongpatra 



Deebuk 

Phang-nga 



Ban Bo San 



Tanpiyaissara 

meeya 



Satree Phang- 
nga 



Satree Phang- 
nga 



Suthipong/ 

Manas 



Satree Phang- 
nga 



Satree Phang- 
nga 



Apinya/ 

Sirikarn 



Jintana/ 

Janoen 



Jurairat/ 

Anonopatra 



Lawin/ 

Anonopatra 



Anyarat/ 

Janoen 



Jarin/ 

Janoen 



Piyawat/ 

Anonopatra 



Prapa/ 

Chalono 



BanTha Noon 



Thesaban Ban 
Thai Chang 
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